BAB^-CONF-06/017 
SLAC-PUB-11991 
July 2006 



Measurement of cos 2/3 in — > D^*^^h^ decays with a time-dependent 
Dalitz plot analysis of — ^ i^^7r''"7r~ 



The BABAR Collaboration 
February 7, 2008 



Abstract 

We report a preliminary measurement of cos 2/3 in D^*^^h^ decays with a time-dependent 

Dalitz plot analysis of — > KgTr^Tr~, where is a light neutral meson such as vr'', r/, t]' or w. 
The strong phase variation on the Dalitz plot allows the access to the angle /? with only a two-fold 
ambiguity (/? + vr). Using 311 x 10^ BB pairs collected at the BABAR detector, we obtain cos 2/3 = 
0.54 ± 0.54 ± 0.08 ± 0.18, sin 2/? = 0.45 ± 0.36 ± 0.05 ± 0.07, and |A| = 0.975l[!:[J^^ ± 0.012 ± 0.002, 
where the first errors are statistical, the second are experimental systematic uncertainties, and the 
third are the signal Dalitz model uncertainties. This measurement favors the solution of /3 = 22° 
over 68° at an 87% confidence level. 
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1 INTRODUCTION 



The Standard Model of electroweak interactions describes charge-parity (CP) violation as a conse- 
quence of an irreducible phase in the three-generation Cabibbo-Kobayashi-Maskawa (CKM) quark- 
mixing matrix pj. In this framework, the CP parameter sin 2/3 can be measured by examining the 
proper-time distribution of neutral B decays to CP eigenstates. This parameter has been measured 
with a high precision by the 5-factories using final states containing a charmonium and a neutral 
kaon 2 . The current average from S-factories is sin 2/? = 0.685 =b 0.032 which leads to a 
four-fold solution of the angle /? = 22°, 68°, (22° + 180°), and (68° + 180°). The ambiguity can 
leave possible new physics undetected even with very high precision measurements of sin 2(3. 

Analyses have been attempted to resolve the (/?, 7r/2 — /?) ambiguity using a time-dependent 
angular analysis in — > J /iI)K*^{K^t:^) decays In this analysis we use a new method proposed 
by Bondar et al. 0, which uses B'^ D^*^^h^ decays with a time-dependent Dalitz plot analysis 
of — > KgTr^7r~ , where is a light neutral meson such as tt^, r]^'\ and lo. This method takes 
advantage of the varying strong phase on the — > K'^tt^tt^ Dalitz plot to resolve the ambiguity 
of the phase 2/3 from the sin 2/3 measurements alone. The Belle Collaboration has recently reported 
a measurement using this technique 0; they obtained cos 2/3 = l-87lo;53lo;32 and determined the 
sign of cos 2/3 to be positive at a 98.3% confidence level. 

The leading order diagram of B'^ — > D^*^'^hP is color-suppressed, as shown in Figure ^ The 
next order diagram is suppressed by 0(sin^ ^Cabibbo)- There are no penguin diagram contributions. 
A sizable new physics effect due to supersymmetry without R-parity is possible in b ^IJd decays, 
while the Standard Model uncertainty is relatively small [Jj. Interference between B^ D^hP and 
via mixing occurs when and decay to a common final state, such as 

K°7r+7r-. 




Figure 1: Leading diagrams for B^ D°h° decays. 

Assuming CPT symmetry is conserved and the decay rate difference AF is negligible, the time 
evolution function for a state that is known to be a B^ at a time t = ttag can be expressed as 

l^phys(*)) = e-r^*/2 p^o^ cos(AmAt/2) + d\B^) sin(AmAt/2)] , (1) 

where F is the average decay rate of the two mass eigenstates of B^ meson, Am is the mixing 
frequency, At = t^ec — *tag is the time difference between B^ decay time tree and itag, and q/p is the 
ratio of \B^) and \B^) coefficients in B^ mass eigenstates. Neglecting CP violation in B^ mixing, we 
assume \q/p\ = 1. Expressing the decay amplitude of the decay chain B^ — > D^h^ — > [K^tt^t:' 
as ^4^ = AA^ and similarly for B^ as Aj = AA^, the decay probability of a neutral B meson in 
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an T{4:S) system can be shown to be 



V± = -e-r^*|A|2 



|A^|2 + |A|VdI') T (|%p-|Ap|^^|2)cos(AmAt) 

± 2\X\7]ho{-l)^lm(e-^''^ADA*^)sm{AmAt)\ , (2) 



qA— 

where the upper (lower) sign is for (i?*^) -tagged events, A = — 2i/3 is the phase of q/p, rj^o is 

the CP eigenvalue of , and L is the orbital angular momentum in the D^*^^h^ system. In the case 
of — > D*^h? (where is a pseudoscalar) with D*^ — > D^ir^, L = 1 and two additional factors 
need to be considered: the angular momentum in D*^ — > D^tt^ (L' = 1) and the CP eigenvalue of 
the soft vr*^ from D*^ decay {-q^o = —1) 0. The decay amplitudes Ao and A^ can be expressed as 
a function of two Lorentz invariant variables = (Pxg +P7r+)^ ^^'^ = (Pk'^ +P7r-)^- That is, 
Ad = A£){'m'^,m?_) and A^ = A£){m?_,m^). Here we have assumed that CP is conserved in the 

decay and neglected mixing. 

In the last term of Equation^, one can rewrite 

Im(^e-2*^^DA^) = Im(ylfl^^)cos2/3-Re(^i)A;^)sin2/3 , (3) 

and treat cos 2/3 and sin 2/3 as independent parameters. 

In this analysis, we use an unbinned maximum-likelihood method to fit for cos 2/3, sin 2/3 and 
|A|, and use a parameterized Monte Carlo method based on the observed data to estimate the 
confidence level of cos 2/3 being positive. 



2 THE BABAR DETECTOR AND DATASET 

This analysis is based on 311 x 10^ BB pairs collected on the T{4S) resonance during 1999-2006 
with the BABAR detector at the PEP-II storage ring. A sample of 23 fb^^ collected at 40 MeV below 
the T{4S) resonance and a number of signal and generic simulation samples based on Geant4 |10j 
are also analyzed to optimize the event selection and to study background properties. 

The BABAR detector is described in detail elsewhere |llj . Charged-particle trajectories are 
measured by a five-layer double-sided silicon vertex tracker and a 40-layer drift chamber located 
within a 1.5 T solenoidal magnetic field. Charged hadrons are identified by combining energy- 
loss information from the tracking system with the measurements from a ring-imaging Cherenkov 
detector. Photons are detected by a CsI(Tl) crystal electromagnetic calorimeter with an energy 
resolution of as/E = 0.023(S/ GeV)-i/^ 0.014. The magnetic flux return is instrumented for 
muon and identification. 



3 EVENT RECONSTRUCTION AND SELECTION 

In this analysis, we reconstruct B^ decays to D^h^, where = 7r^(77), 7/(77, 7r"''7r~7r'^), ^/'(vr^vr^r/), 
and a;(7r+7r-7r°), and 5° D*^{^ D°TT^)h°, where /i° = 7r°(77) and 1/(77). The is recon- 
structed in i^gVr^vr" mode. 

A charged track must be reconstructed in the drift chamber, and, if it does not result from a 
Kg decay, it must extrapolate back to within 1.5 cm of the nominal interaction point in the plane 
transverse to the beam and 10 cm along the beam. A cluster found in the calorimeter that is not 
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associated with a charged track is considered a photon candidate if its shower shape is consistent 
with a photon and its energy is greater than 30 MeV. 

The TT^ candidates are reconstructed by combining two photon candidates with the 77 invariant 
mass in the range 110-160 MeV/c^ if used in D*^ — > D^ir^ reconstruction, or 115-150 MeV/c^ if 
used in B'^ — > D^*^^7r^ reconstruction; for the latter, each of the two photons is required to have an 
energy greater than 50 MeV. For rj — > 77, the photon candidates must both have an energy greater 
than 100 MeV and the photon pair must have an invariant mass within 40 MeV/ of the nominal 
r] mass J2j and have a momentum greater than 200 MeV/c in the laboratory frame. If the 77 ^ 77 
candidate is later used in a D*^ri candidate, the mass window is tightened to 33 MeV/c^. The rj 
candidate is removed if the invariant mass of one of the photons and another photon in the rest 
of the event is within 6 MeV/c^ of the nominal tt^ mass. For r] — > vr+vr^vr'^, the invariant mass of 
the candidate is required to be within 9 MeV/c of the nominal rj mass. An ry' candidate is formed 
by combining an rj candidate with two pions. The invariant mass must be within 8 MeV/c^ of 
the nominal r]' mass. An u! candidate is formed by combining 7r+7r~7r''. The invariant mass of the 
three-pion candidate is required to be within 18 MeV/c^ of the nominal lo mass. The vr*^ candidate 
used in u) reconstruction is required to have a momentum greater than 200 MeV/c in the laboratory 
frame. Except for to, all are fitted with their mass constrained at the nominal value. 

A Kg candidate consists of a vertexed pair of oppositely-charged tracks with an invariant mass 
within 12 MeV/c^ of the nominal mass with a probability greater than 0.1%. The Kg flight 
distance must be greater than three times the estimated uncertainty, and the angle between the 
flight direction and the vertex displacement from the beam spot in the transverse plane must satisfy 
cos 6* > 0.992. 

A candidate consists of a pair of oppositely-charged tracks and a Kg candidate. The 
invariant mass, mjjo, must be within 60 MeV/c^ of the nominal mass. The m^jo resolution 
is approximately 7 MeV/c^. We retain the sideband for later use in the fit. We then fit the 
kinematic parameters with both and K^ constrained at their respective nominal mass. These 

candidates are combined with a vr*^ to form a D*^ candidate. The invariant mass is required to 
be within 3.0 (2.8) MeV/c^ of the L>*° nominal mass in D*^ir^ {D*^r]) reconstruction. 

Eventually we build a B^ candidate combining a , rj, uj or rj with a or a D*'^ candidate. We 
fit the B^ decay vertex requiring that the production vertex is consistent with the beam spot in the 
transverse plane. The energy-substituted mass tties = -\/(s/2-|-po-pB)^/-E'o — |p_Bp is required 
to be greater than 5.2 GeV/c^, where s is the squared center-of-mass (cm.) energy, (Eq^ po) and 
{Eb^ Pb) are the four-momentum of the initial state e^e~ and the B candidate, respectively. The 
energy difference Ai? = E*^ — -Ebeam' evaluated in the cm. frame, must be between ±80 MeV 
(±40 MeV) for events with /i*^ 77 (/i*^ — > tx^tx^tx^ and rj — > ix'^ix^rj). 

The majority of the background is from qq continuum events. We suppress them by requiring 
the normalized second order Fox- Wolfram moment K2 ^B] to be less than 0.5 and |cos0t| less 
than 0.9, where Ot is the angle between the thrust of the B candidate and the thrust of the rest 
of the event. We further suppress the continuum events by a Fisher discriminant formed from 
the following five variables: cos Ot^ the B flight angle in the cm. frame, total event sphericity, 
total event thrust magnitude, and the ratio of two moments L2/L0, where Lj = X^jP^I cos^^l', 
summed over the remaining particles j in the event, where Q*^ and p*^ are the angle with respect 
to the B^ thrust and the momentum in the cm. frame. For D^bj events, two variables are added 
to take advantage of the polarization of the uj: the angle between the B flight direction and the 
normal to the three-pion plane in the uo ■t:^'k~'k^ rest frame, and the angle between one pion 
in the rest frame of the remaining pion pair with respect to the direction of the pion pair. The 
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Fisher coefficients are calculated using off-resonance data and simulated signal event samples. The 
optimum selection value is determined mode by mode by maximizing the signal yield significance 
using simulated signal and generic background events. 

For vr", one major background source is the color-allowed decay — > D^p^ , which 

has a branching fraction approximately 50 times larger. For events reconstructed as a D^-k^, the 
D^p'^ contribution peaks in AE below the selection region and only a small number of events is 
selected. However, for events reconstructed Z)*°7r°, the final state D° TT^TT^ is very similar to 
-D'^(7r°7r^)p+. Therefore this background has a mean AE near zero. We veto B'^ D^p^ events 
by rejecting D*^tt^ candidates if the vr*' candidate combined with any other charged pion in the 
event can form a p'^ candidate with an invariant mass within 250 MeV/c^ of the nominal value, and 
subsequently form a B^ candidate by combining with the candidate. The requirements for the 
B^ candidate are rriES > 5.27 GeV/c^ and |Aii^| < 100 MeV. Finally, we only retain events with 
decay time difference |At| < 15 ps and the estimated uncertainty a^t < 3.6 ps. If there is more 
than one candidate in the event, the one with a more signal- like Fisher discriminant is selected. 

We use a two-dimensional (mESj^^D") probability density function (PDF) in an unbinned- 
maximum-likelihood fit to separate three types of events: (1) signal: a single Gaussian in tties and 
a Crystal Ball function |15j in m^io; (2) combinatorial background with a real D^: an Argus jl4j 
function in rriES and a Crystal Ball function in m£,o; (3) combinatorial background with a com- 
binatorial D^: an Argus function in uies and a first order polynomial in itido. The Crystal Ball 
parameters for m£,o in components (1) and (2) share the same values, as does the Argus parameter 
in (2) and (3). 

The results of the fit are shown in Figures |21 and 13 the yields are shown in Table ^ We merge 
the D^r] and D^-q' samples, as well as the D*^7:^ and D*^r] samples. The Dalitz distributions in the 
signal region and tit-es sideband are shown in Figure|3 There are irreducible backgrounds that peak 
in both m-Es and mjjo, which cannot be discriminated against with our PDF. The majority of this 
peaking background is from B~^ — > D^*^'^p~^ . We estimate the amount of the peaking background 
using the simulated generic Monte Carlo samples. The number of signal events after subtracting 
the peaking background is shown in the last row in Tabled 



Table 1: Signal event yields. 



Decay mode 






ri' 






Total 


Raw peak yield 


175 ± 17 


97 ± 


11 


93± 12 


59 ±9 


424 ± 25 


Peaking background subtracted yield 


168 ± 19 


87 ± 


12 


82± 13 


47 ±9 


384 ± 28 



4 DALITZ PLOT MODEL 

The decay amplitude is determined from an unbinned maximum-likelihood fit to a high-purity 
sample of D*~^ — > D^7r~^ decays. We use the isobar formalism described in ^] to express Ajj 
as a sum of two-body decay matrix element (subscript r) and a non-resonant (subscript NR) 
contribution. 



11 



(D 

o 

glOO 



BaBar 

preliminary 



5.22 



5.24 




5.26 5.28 
niEs (GeV/c^) 



5.26 5.28 
niEs (GeV/c^) 




5.26 5.28 
niEs (GeV/c^) 



5.26 5.28 
niEs (GeV/c^) 



Figure 2: Distributions of ttt-es for (a) D'^tt^, (b) D^r]^'\ (c) and (d) D*^Tr'^,rj for events with 
m£)0 within 14 MeV/c^ of the nominal value. Curves are: (solid) overall PDF projection; (dotted) 
background (including peaking) PDF; (dashed) contribution from background with fake D^. 



where each term is parameterized with an amplitude and a phase (pr ■ The function Ar (m^ , ) 
is the Lorentz-invariant expression for the matrix element of a meson decaying into i^TgVr^Tr" 
through an intermediate resonance r, parametrized as a function of the position in the Dalitz plane. 

The resonances in the model for are: (a) [^Cgvr"] resonances: K*{892)^, i^Q(1430)~, 
i^2*(1430)-, K*(1410)-, and A"*(1680)-; (b) [K^Tr+] resonances (doubly-Cabibbo suppressed): 
i^*(892)+, ifo*(1430)+, and i^2*(1430)+; (c) [vr+vr-] resonances: p(770), w(782), /o(980), /o(1370), 
/2(1270), p(1450) and two scalar resonances a and a'. For p{770) and p(1450) we use the func- 
tional form suggested in JHI j while the remaining resonances are parametrized by a spin-dependent 
relativistic Breit-Wigner distribution. The means and widths of the resonances are taken from the 
PDG ^2j, except for a and a' , which are obtained from the Dalitz plot fit. The inclusion of a and 
a' significantly improves the Dalitz plot fit quality. However, since these two scalars are not well 
established, we consider the systematic effect of using a model without them. More details about 
the Dalitz plot model and parameters can be found in [El- We neglect the detector resolution in 
the Dalitz plot model because the resolution (~ 4 (MeV/c'^)^) is much smaller than the resonance 
widths. Only a;(782) has an intrinsic width comparable to the mass resolution. We increase its 
width artificially in the systematic study. 
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Figure 3: Distributions of rrij^o for (a) D^tt'^, (b) D'^ri^'\ (c) D^uj and (d) D*^Tr^,r] for events with 
mES > 5.27 GeV/c^. Curves are: (solid) overall PDF projection; (dotted) background (including 
peaking) PDF; (dashed) contribution from background with fake D^. 



5 TIME-DEPENDENT ANALYSIS 

We model the time-dependent Dalitz plot distribution in a PDF that consists of four components: 
signal, background with a real D^, background with a fake and background that peaks in 
both mES and m £)0 . The (m-ES , m£)0 ) model for the first three components has been described 
in Section 131 The peaking background component shares the same (weSj'^do) shape with the 
signal component. The background fractions are determined from a fit to (w-eSj ^do) distributions 
and from generic Monte Carlo samples (for peaking background) for each mode group and 
each tagging category. Each event is assigned signal and background probabilities based on the 
two-dimensional PDF. 

The time-dependent Dalitz model for signal is based on Equation((2j. We modify the equation 
to take into account mistagging and imperfect At reconstruction, following the methods used in our 
other time-dependent analyses ^S]) i-e., an additional factor (1 — 2w) is added to the cos(AmAt) 
and sin(AmAt) terms, and the equation is convolved with a three-Gaussian At resolution function. 
There are six tagging categories with different mistag fractions w. We also allow the w of each 
category to be different for and B^ tags. The means and widths of the core and the second 
Gaussian are parameterized with scale factors multiplied by cJAt- The mean and width of the third 
(outlier) Gaussian are fixed at ps and 8 ps, respectively. The mistag rates and the resolution 
function are determined from control samples of B^ — > D^*^tt, p,ai decays. Most of the resolution 
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Figure 4: Dalitz distributions for (a) -B'^-tagged and (b) B^-taggcd events in the signal region, 
TTT-ES > 5.27 GeV/c^, and (c) events in the niES sideband tjies < 5.26 GeV/c^. In ah cases, the 
mass is required to be within 20 MeV/ of the nominal value. 

function parameters are consistent among the six tagging categories, except for the core Gaussian 
mean and scale factor, where the Lepton tagged sample is significantly different from others. We 
allow these two parameters to be different for Lepton tag. 

The model for the background with a fake is determined from the sideband data. The 
At model consists of a prompt component and a exponential decay component with an effective 
lifetime. The resolution function is a Gaussian whose mean and width are scaled by cr^t, plus an 
outlier Gaussian. The mean of the core Gaussian and the fraction of the prompt component are 
allowed to be different between the Lepton tag and the other tags. 

The Dalitz distribution for background is modeled by an incoherent mixture of several reso- 
nances and a phase-space distribution, 

V{ml,m'i) = |aNRp + ^|a^P|A(m^,m2)p . (5) 

r 

We find that the model describes the D° sideband data well if we include K*(892)-, K*(892)+, 
Kq(1430)~, p(770), p(1450) and a resonances in the model. We also check that the Dalitz distri- 
bution is independent of the tagging category, the flavor tag, and At. 

Based on a study using the generic Monte Carlo samples, the background with a real comes 
mostly from cc continuum events. We therefore model the At distribution with a prompt component 
convolved with a core Gaussian plus outlier resolution function. The parameters are determined 
from the generic Monte Carlo sample. 

The Dalitz model for this background is either AD{m^, m?_) or Ajy{m\,rn?_) based on the flavor 
of the tagging side Stag- If Aag is tagged as (B^), the D in the reconstructed candidate is more 
likely to be a {D^)- Since they are not BB events, the mistag rates are not the same as those for 
the signal. However, we don't have reliable mistag values for continuum events. We therefore use 
the mistag rates for signal in the nominal fit and vary them to estimate the systematic uncertainty. 

The peaking background being mostly from charged B decays, the At model is an exponential 
decay with the lifetime fixed at the B~^ lifetime. The Dalitz model is identical to that for combi- 
natorial background with real D^, except that the mistag rates can be different. Again we fix the 
mistag rates to those for the signal, and vary them for systematic uncertainty. 
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In the nominal fit, we allow cos 2/3, sin 2/? and |A| to float and fit to all data samples and 
tagging categories simultaneously. The lifetime and mixing frequency are fixed at the PDG 
values. The fit results are shown in Table [2 where the result for which the sin 2/3 is fixed at the 
world average and |A| at one is also included. We also allow the ttt-es shape and background fractions 
to float in the flt. We find no significant difference in either the central values or the statistical 
uncertainties. The projections on the Dalitz plot variables are shown in Figure El and are compared 
with the distributions described by the model. Figure IHl shows the time-dependent CP asymmetry 
for events in Kgp{770) region (|m(7r^7r~) — 0.77| < 0.25 GeV/c^), where the CP asymmetry 

is expected to be enhanced. The apparent asymmetry in Figure IHl is small compared to sin 2/3 
due to the dilution factors from mistagging, background and contributions from non-resonance and 
resonances other than p. 

Table 2: Results of the fits to data. Errors are statistical only. 



Final state 


cos 2/3 


sin 2/3 


|A| 


D*07r0/r/ 


0.4 ± 1.1 
0.3 ± 1.4 


1.0 ±0.5 

-0 1+°-^ 
0.7 ± 1.0 

-U.»_o.9 


1 is+'J-i'^ 

96+°-^^ 
u-^O-o.ie 


Ah 


0.54 ±0.54 


0.45 ±0.35 


0.98 ± 0.09 


All 


0.55 ±0.52 


0.685 (fixed) 


1 (fixed) 




Figure 5: Projection on (a,d) m?,o +j (b,e) _, and (c,f) ?7i?,o _ for (a,b,c) iJ^-tagged and 

(d,e,f) S'^-tagged events separately, in the signal region (mss > 5.27 GeV/c^, \m£,o — m™^| < 
20 MeV/c^). Points with error bars are data; histograms are from the PDF. 
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Figure 6: Asymmetry distribution for the events in Kgp{770) region. The curve is the result 

of the PDF. 

6 SYSTEMATIC STUDIES 

Dependence on the choice of Dahtz plot model in signal is expected to be one of the largest 
systematic uncertainties. We estimate the systematic uncertainty by comparing the nominal Dalitz 
model and an alternative model where the two scalar resonances a and a' are removed. We generate 
300 toy datasets using the parameters from the nominal fit; each toy dataset has 50 times the data 
size. We fit to each toy dataset using both nominal Dalitz model and the alternative Dalitz model. 
We find that both sin 2/3 and cos 2/3 shift significantly. We take the quadratic sum of the mean and 
RMS of the distribution of the difference between the two models as the systematic uncertainty due 
to Dalitz model uncertainty. The detector resolution on the Dalitz plot is neglected. Only u;(782) 
has an intrinsic width comparable to the mass resolution. We increase its width from 8.5 MeV to 
10 MeV and find no significant change in the results. 

As described in EquationQ, the background Dalitz model is described by an ad hoc incoherent 
mixture of several resonances and a phase-space distribution. Alternatively, we use a background 
model containing only K* and a phase-space distribution to describe the sideband Dalitz dis- 
tribution. This model describes the data rather poorly. However, the changes to the final results 
are quite small. 

We vary the lifetime and mixing by their uncertainty quoted in the PDG [213], and other fixed 
parameters by their statistical uncertainty in the fits to control samples, to estimate the systematic 
uncertainties. The parameters include the amplitudes and phases of the Dalitz model, rriEs, niDO 
and background At distributions, background fractions, mistag rates and resolution functions. In 
addition, we vary several resolution function parameters that were fixed in the fit to the control 
sample: outlier bias from —2 to +2 ps, outlier width from 4 to 12 ps, and the second Gaussian 
scale factor from 2 to 5. 

The mistag rates in both peaking and combinatorial background with a real are the same 
as the signal mistag rates in the nominal fit. We vary the mistag rate of each tagging category by 
±30% for both backgrounds to estimate the systematic uncertainties. 

We fit a two-dimensional third-order polynomial to signal Monte Carlo samples to parametrize 
the reconstruction efficiency variation over the Dalitz plot. We vary the parameters by their one- 
sigma statistical uncertainty and the variation in the final answer is again negligible. If we simply 
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assume the efficiency is a constant across the Dahtz plot, the changes to the results are still quite 
small and we treat these differences as systematic uncertainties. The systematic uncertainties are 
summarized in Table |31 



Table 3: Summary of systematic uncertainties. 



Item 


cos 2/3 


sin 2/3 


1^1 


Signal Dalitz model 


0.184 


0.073 


0.002 


Signal Dalitz parameters 


0.068 


0.026 


0.006 


Background Dalitz parameters 


0.002 


0.001 


0.002 


Background Dalitz model 


0.004 


0.006 


0.008 


Am 


0.003 


0.002 


0.000 




0.003 


0.001 


0.000 


tb+ 


0.003 


0.000 


0.000 


Mistag, resolution, etc. 


0.043 


0.043 


0.002 


Peaking background fraction 


0.020 


0.018 


0.005 


Mistag in combinatorial background with 


0.002 


0.001 


0.001 


Mistag in peaking background 


0.002 


0.001 


0.001 


Dalitz plot efficiency 


0.002 


0.001 


0.000 


Total (non-Dalitz-model) 


0.083 


0.054 


0.012 


Total 


0.202 


0.091 


0.012 



7 RESULTS 

We have measured the CKM phase cos 2/3 and sin 2/3 using a time-dependent Dalitz plot analysis 
of —>■ KgTT^7r~ decays in D^h? decays. We obtain 

cos 2/3 = 0.54 ±0.54 ±0.08 ±0.18 (6) 
sin 2/3 = 0.45 ± 0.35 ± 0.05 ± 0.07 (7) 
|A| = 0.975l{]:|]^^ ± 0.012 ± 0.002 , (8) 

where the first error is statistical, the second is experimental systematic uncertainty, and the 
third is the signal Dalitz plot model uncertainty. The statistical correlation between cos 2/3 and 
sin 2/3 is 7%, and less than 5% between |A| and cos 2/3 or sin 2/3. The result is consistent with 
sin2/3o = 0.685 ± 0.032 and cos2/3o = ± y^l - sin^ 2/3o = ±0.729 obtained from high precision 
measurement using to charmonium decays, and consistent with no CP violation in B decay 
([A[ = 1). If sin 2/3 is fixed at 0.685 and |A| at one in our analysis, we obtain 

cos 2/3 = 0.55 ± 0.52 ± 0.08 ± 0.18 . (9) 

This result allows one to distinguish the two possible solutions of angle 2/3o. We generate 1500 
parametrized simulation samples of the same size as the data sample, where we use sin 2/3 = sin 2/3o, 
cos 2/3 = I cos2/3o| and |A| = 1. We then fit to each sample with cos 2/3 as the only free parameter, 
and use a two-Gaussian function to fit to the distribution of the 1500 results. We repeat the 
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same exercise using cos 2/3 = — |cos2/3o| to generate simulation samples and obtain another two- 
Gaussian distribution The distributions are shown in Figure [7| We define the confidence 
level (CL) at which the cos 2/3 = — | cos 2/3o | solution is excluded when we observe cos 2/3 = x as 
h-\-{x) /[h-\-{x) + We calculate the CL for cos 2/3 = 0.55, 0.35 and 0.75 to account for the 
systematic uncertainty and use the smallest value, 87%, as the final result. 




cos2p 

Figure 7: Distribution of cos 2/3 obtained from two sets of 1500 simulated experiments of the same 
size as the data sample, as described in the text. Distribution in solid dots (open squares) is for 
samples generated with cos 2/3 = |cos2/3o| (— | cos 2/3o|). Solid (dashed) curve is the corresponding 
two-Gaussian function /i+(x) (/i_(x)). Three vertical arrows indicate the central value of cos 2/3 
and plus/minus systematic uncertainty. 



8 CONCLUSIONS 

We have studied the time-dependent Dalitz distribution in D^*^^h^ decays and determined 

the CP asymmetry parameters sin 2/3, cos 2/3 and [A|. The results are consistent with the Standard 
Model expectations. Assuming sin 2/3 is equal to sin2/3o found in to charmonium analyses 
and no CP violation in B decays, we determined that the solution cos 2/3 = — sin^ 2/3o is 

excluded at an 87% confidence level. 
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